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ABSTRACT 



To better conceptualize educational effectiveness, a 
comprehensive model (Creemers 1994) that takes as a point of departure the 
differences in the learning results of students was developed. To explain the 
differences in outcomes, this paper introduces more formal 
characteristics--consistency , cohesion, constancy, and control--for 
effectiveness in the comprehensive model . Following the introductory section 
and a discussion of the advances and problems in educational effectiveness 
research in section 2, section 3 explains the development of a comprehensive 
model and concludes that formal characteristics exist to some extent. Section 
4 examines teachers' visions about education at the classroom level, school 
level, and within the wider educational context. The concept of vision 
combines ideas, opinions, beliefs about instruction, schools, and education 
in general . Visions are strongly related to concrete behavior in the 
classrooms and schools. Vision, in a comprehensive model for educational 
effectiveness, is related to factors for educational effectiveness and the 
formal criteria. One figure and four tables are included. (Contains 42 
references.) (LMI) 



★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★a-* 



Reproductions supplied by EDRS are the best that can be made 
from the original document . 



★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★it* 



X 



m 

VO 

m 




>5 

<\ 



do 




VISIONS THAT WORK 

further elaboration of the comprehensive model 
of educational iffdctiveness 



Bert P.M. Creemers 

Groningen Institute for Educational Research 

GION 

P.O. Box 1286 
9701 BG Groningen 
The Netherlands 

Tel: +31 50 3636635 Fax: +31 50 3636670 
E-mail: C.M.A.Lenderink@ppsw.rug.nl 

Paper prepared for 

the American Educational Research Association 

BEST COPY AVAILABLE 

Chicago, 27 March, 1997 



U S. DEPARTMENT OF EDUCATION 

Oflce ol Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

Ef This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to improve 
reproduction quality. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

4 * 




Points of view or opinions stated m this docu- 
ment do not necessarily represent official 
OERl position or policy 



.... .J 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



f 



1 Introduction 

In order to conceptualize educational effectiveness better we have developed a 
comprehensive model of educational effectiveness (Creemers, 1994). The model takes as 
point of departure the differences in learning results of students. It points at the importance 
of learning in schools, including learning processes by students and instructional processes 
that take place in classrooms and schools. Taking into account the research evidence about 
the effectiveness of textbooks, grouping procedures and especially teachers’ instructional 
behaviour factors are formulated at the classroom level that explain differences in student 
outcomes. The levels above the classroom, like the departmental level, the school level and 
the above school level are seen as providing conditions for what happens at the classroom 
level. From a theoretical point of view all the factors in the model can be allocated to time 
for learning, opportunity to learn and quality. With respect to time and opportunity we 
distinguish the time and opportunity offered by education and the use of time and oppor- 
tunity by students. We suppose that the latter is more directly related to student outcomes. 

In order to explain the differences in outcomes we introduce more formal 
characteristics for effectiveness in the comprehensive model. These concern the 
relationships between the factors of instruction (textbook, grouping procedures and 
teaching behaviour), the stability of factors over time and mechanisms to ’impose’ the 
factors. The idea behind these formal characteristics is that the influence of the factors at a 
certain level and between factors at different levels can be enforced or take place by the 
fact that these factors are pursued for a longer period of time and are in line with each 
other. Formal characteristics concern the consistency between the factors of different 
educational elements, the cohesion within the school and system to pursue certain factors 
over time, the constancy by which students are exposed to these factors during their school 
life and the control mechanisms to enhance certain factors. 

Past research into factors in the model does not provide definite evidence for their 
contribution to effectiveness. In our research programme we are currently trying to pursue 
the influence of the formal characteristics, next to the factors at different levels. The main 
question is whether consistency between factors at a certain level or between levels and 
cohesion and constancy over time and a system that controls the implementation of the 
factors influence educational effectiveness. Section 3 presents the results of a project, so 
far restricted to consistency. The project concludes that formal characteristics exist to some 
extent. This conclusion stems from empirically based configurations of factors in schools 
and classrooms. However, there is another way of looking at the formal characteristics that 
could be more productive. 

In this approach we take as a point of departure the fact that educational 
professionals have a vision about education. We expect that such a vision is related to 
what they actually do. Although the same holds for other people like students, parents, and 
civil servants involved in education, we concentrate on the teachers as the key persons in 
education. We will develop the concept of vision at the different levels of education like 
the classroom, the school and the educational context and in this way we will relate it to 
factors and formal criteria for educational effectiveness. We will address the question as to 
whether the concept of vision is an alternative idea for a comprehensive model for educa- 
tional effectiveness, an addition to the model or a rephrasing of elements already included 
in the model. 
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2 Advances and problems in educational effectiveness 

After the early enthusiasm in the United States for school effectiveness and school 
improvement, the conceptualization, research designs and results were strongly criticized. 
In the second half of the eighties educational research in the States already moved to other 
topics in education but a strong attention for theory and research in educational 
effectiveness came up in other countries like the UK, Australia, the Netherlands and Hong 
Kong. Results of that international attention for educational effectiveness can be found in 
the ten volumes of the journal of School Effectiveness and School Improvement, related to 
the International Congress of School Effectiveness and School Improvement. Handbooks 
and state of the art reports were published about the advances in research, theory and prac- 
tice with respect to educational effectiveness (Cheng, 1996; Creemers, 1994; Reynolds et 
al., 1994; Sammons, Hillman & Mortimore, 1995; Scheerens, 1992; Scheerens & Bosker, 
1997; Stringfield & Herman, 1996; Townsend, 1994). These show that theory and research 
in the area of educational effectiveness have progressed considerably recently. 

Theoretical issues 

Originally educational effectiveness was mainly a collection of factors that could be found 
in effective schools and less or not in ineffective schools. Later on theoretical models were 
developed that put together an increasing list of these factors in a more or less logical 
order. Especially in Europe attention was paid to the further elaboration of theories and 
models, which included the addition of factors that could improve the model. Also, a 
reconstruction in theoretical terms of already existing theories was pursued. This orienta- 
tion put forward concepts which in relation to each other could explain differences 
between effective and ineffective schools. In order to increase the explanatory power of 
the theory new concepts were included as well. 

For the development of theories it was necessary to determine what effectiveness in 
the end was. That has lead to further theorizing about quality and equity in education and 
resulted in the determination of other objectives of education, more attention for non-aca- 
demic goals and the relationship between various domains of educational objectives with 
respect to effectiveness. Also the concept of differential effects, different outcomes of 
education for different groups of students, came up. In order to find a conceptual basis for 
educational effectiveness the focus of attention shifted from factors merely related to 
effective schools and classrooms to what explains learning outcomes. Especially the 
question as to how factors related to effective schools and classrooms could be 
reconceptualized in terms of causes for effects received a lot of attention. 

Relating effective factors to theories about instruction is one way to increase the 
theoretical level of educational effectiveness. These theories in their turn are related to stu- 
dent learning as well. Another way is to look at schools as organisations and to relate 
effective factors to organisational theories, although in the end these theories are related to 
processes and outcomes too. A further step forward was to emphasize student learning at 
the classroom level in effectiveness theory and to conceptualize the contribution of the 
level above the classroom to what happens at the classroom level. That has resulted in a 
more refined specification of factors at departmental, school and other levels, especially 
factors that -could be related to processes at the classroom level (Cheng, 1996; Creemers, 
1994; Creemers & Reezigt, 1996; Scheerens, 1997; Slater & Teddlie, 1992; Stringfield & 
Slavin, 1992; Teddlie & Stringfield, 1993). 




Empirical issues 

Although the improvement that got the most attention in last years was statistical analysis 
in effectiveness research, other issues related to research are equally important. Recent 
research pays more attention to various objectives and the measurement of student 
outcomes. More than in the past the importance of ways to measure instructional 
effectiveness at the classroom level and factors at the other levels reliably and valid is 
emphasized. The development of instruments is based more strongly on theoretical ideas 
mentioned above and depends less on just a collection of more or less common sense ideas 
about what might be related to the effects of education. Furthermore research pays 
attention to issues of stability, size and relevance of effects (Goldstein, 1997; Scheerens & 
Bosker, 1997; Thomas, Sammons, Mortimore & Smees, 1997; Van der Werf & 
Guldemond; 1996). 

Effectiveness research and improvement of education 

Although educational effectiveness could be conceived as an academic research program, it 
started out and still is a program for the improvement of education as well. In educational 
practice, theory and research find a place to test and prove the ideas. Educational practice 
can benefit from theoretical insights and contribute to the further elaboration of theory and 
research. In the last decade, attention for the complicated process of the implementation of 
research results into practice and the way educational practice can contribute to and 
influence the research program has grown. A new way of merging educational effective- 
ness research and school improvement can contribute to further advances in the 
methodology of research and promote longitudinal studies, case studies and real life 
experimental studies (Creemers & Reezigt, 1997; Gray, Reynolds, Fitz-Gibbon & Jesson, 
1996; Hopkins, 1996; Hopkins & Lagerweij, 1996; Reynolds et al., 1996). 

Looking at the advances, the research programme and the related school improvement 
seem to have greatly improved. However, not only optimistic conclusions can be drawn. 
There are also some problems. 

First, it seems that the advances are made particularly in the field of theory 
development, designs, and research techniques including statistical procedures. 
Although the major proportion of variance in student outcomes is accounted for by 
background factors like students’ ability, motivation and the home environment, it 
is still hard to relate the small proportion of variance accounted for by schools and 
classrooms to specific factors discussed in the literature. 

Sometimes research detects a factor with significant effects and then a specific 
research project is quite positive with respect to the contribution of that factor. 
However, many factors do not seem to be very stable in their effects. This means 
that not only outcomes of classroom and schools are unstable, but the factors which 
can explain the outcomes are quite unstable too. Research has paid attention to 
stability issues with respect to the effects of education, but more important for the 
further theory development and the empirical validation is the stability of the 
factors that explain the effects. 

Research is full of anomalies and contradictory evidence. For example, homework 
is sometimes positively related to outcomes (Walberg & Paschal, 1995) and 
sometimes negatively (Kuyper & Swint, 1996). Sometimes the school level explains 



unexpectedly more variance than the classroom level, which we expect to 
contribute more to effectiveness. When more effectiveness criteria are taken into 
account the picture becomes even more diverse. 

The quality of research has not improved sufficiently, maybe because the 
improvement in theory was not accompanied by further development of research. 
An explanation could be the funds available for research or the fact that new 
insights become available at a time when research studies have already been 
designed. Especially the design of longitudinal studies takes time and subsequently 
the study has to be carried out according to the research plan, even though in the 
meantime new ideas come up. Such studies therefore are criticized even before they 
are published. The effectiveness research basis in the Netherlands for example is 
formed by cohort studies which are not designed completely in line with new 
insights about effectiveness. Nevertheless they are still used as an empirical basis 
for effectiveness research. This situation may explain some of the disappointing 
results. 

When the advances are compared with the problems, the conclusion may be drawn that the 
cleverer we are in theory and conceptualization, the less we succeed in finding empirical 
support for our position. A better conceptualization and theoretical interpretation of 
educational effectiveness alone cannot bring things further. Research is not consistently 
designed on the basis of a theoretical framework but is often based on the availability of 
datasets which offer only some factors related to effectiveness. Still, several studies were 
published which at least proved our understanding about the effectiveness of education. 
Although there are complaints about the quantity of research studies, there are 
internationally many interesting studies (Hill & Rowe, 1996; Cheng, 1996; Townsend, 
1997). Generally they prove the importance of the classroom level and bring forward 
factors that more or less resemble factors that were already discerned. 

The problem however is that theory and research are not in line with each other. 
The theory, departing from theories about learning, instruction, and organisation, tries to 
put together in a comprehensive way the factors that are important for effectiveness. But 
research is not able to provide empirical support for considerable parts or components of 
the specific theories. As a result both theory and research do not provide a firm basis 
about what works in education and why. 

It was and still is a purpose of the comprehensive model of educational 
effectiveness (Creemers, 1994) to improve this situation and to enhance further 
development of the theory together with empirical research. Theory and research should be 
more in line with each other, which means that research should be based on theoretical 
considerations and the results of research should lead to further development and 
reinterpretation of the theoretical model. 

3 Development and research of the comprehensive model 

Like other models developed in the nineties (Scheerens, 1992; Stringfield & Slavin, 1992; 
Stringfield, 1994, Slavin, 1996) the comprehensive model distinguishes between levels in 
education (Creemers, 1994). Higher levels are supposed to provide the conditions for what 
happens at the levels below. This means that not just one level induces results but the 
combination, of levels. 

Most models are rather detailed at the classroom level, while the school level often 
shows a more diffuse selection of variables. Factors are listed based on ideas about how 
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With respect to curriculum the selected factors are: 

explicitness and ordering of goals and content, 
structure and clarity of content; 



advance organizers; 

material for evaluation of student outcomes, 



feedback and corrective instruction. 



With respect to grouping procedures they are: 
mastery learning; 
ability grouping; 

WghlyTprad^t^n differentiated material, material for evaluation, feedback and 
corrective instruction. 

With respect to teacher behaviour they are: 

management, orderly and quiet atmosphere; 



homework; 

S'goaT^ttTng (restricted set of goals, emphasis on basic skills, emphasis on 

« of goals and content, advance organizers, making 

use of prior knowledge of students); 

clarity of presentation; 

questioning; . 

immediate exercise after presentation of new content, 

evaluation, feedback and corrective instruction. 



Factors at the school level 

Looking at the wellknown lists of effective school factors (Levine & Lezotte, 1990, 
Scheerens, 1992; Reynolds, 1993), it becomes clear that most f^s m fact r fl h 
indicators of quality of instruction, time and opportunity to learn at classroom 
Because of a lack of research studies that analyse school and classroom leve toget er, i 
hard 'fo 6 say what the contribution of the school factors might be in accounting for studen 
Lvel v^Le when controlling for classroom factor. In any case, 
factors are rather meaningless when not clearly linked to classroom factors (Cree , 
1992). Even if they have an independent effect on student achievement, it is s 1 no 

how this effect comes about and how it should be interpreted. f 

The model defines all school level factors as conditions for classroom level factors. 

This de“es, rics ,he selection of school factors 

and di recti v related to quality of instruction, time or opportunity to learn. Except tor ba 
"such as the presenceof a time schedule and a school curriculum the school factors 
refer to the shared responsibilities and ideas of teachers when they want to work as a team. 

to do with a system of rules and 

about their classroom practice and facilities to implement and to control these rules an 
agreements S factors at the school level are divided in two groups: educa tona 
facmtT that deal directly with what is going on in the classrooms, and organisational 
factors that determine what happens in classrooms from a greater distance. To eI * a " ce 
totality Of instruction at the classroom level, a school should cover both groups of factors. 
In addition, a school should watch over time and opportunity as well. 
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At the school level, the conditions for the quality of instruction with respect to the educa- 
tional aspects are: 

rules and agreements about all aspects of classroom instruction, especially 
curricular materials, grouping procedures and teacher behaviour, and the 
consistency between them; 

an evaluation policy and a system at the school level to check student achievement, 
to prevent learning problems, or to correct problems at an early stage. This includes 
regular testing, remedial teaching, student counselling, and homework assistance. 
With respect to the organizational aspects of the school level, important conditions for 
quality of instruction are: 

a school policy on intervision and supervision of teachers, departmental heads and 
school principals (educational leadership), and a school policy to correct and further 
professionalise teachers who do not live up to the school standards; 
a school culture inducing and supporting effectiveness. 

Conditions for time at the school level are: 

the development and provision of a time schedule for subjects and topics; 

rules and agreements about time use, including the school policy on homework, 

student absenteeism, and cancellation of lessons; 

the maintenance of an orderly and quiet atmosphere in the school. 

Conditions for opportunity to learn at the school level are: 

development and availability of a curriculum, school working plan or activity plan; 
concensus about the ’mission’ of the school; 

rules and agreements about how to proceed, how to follow the curriculum, 
especially with respect to transition from one class to another or from one grade to 
another). 

Factors at the context level 

The same components as mentioned before, quality, time and opportunity to learn can be 
distinguished at context level. Quality regards the following conditions: 
a (national) policy that focusses on effectiveness of education; 
the availability of an indicator system and/or national policy on evaluation/a 
national testing system; 

training and support systems promoting effective schools and instruction; 
funding of schools based on outcomes. 

Time at the context level refers to national guidelines with respect to the time schedules of 
schools and the supervision of the maintenance of schedules. The opportunity to learn at 
the context level refers to national guidelines and rules with respect to the development of 
the curriculum, the school working plan and the activity plan at the school level, for 
example, by a national curriculum. 

Formal criteria: consistency, cohesion, constancy and control 

In addition to the factors at the classroom, school and context levels, more formal 
characteristics can be discerned (Creemers, 1992). Although focussing on the factors, the 
model also makes tentative statements about their joint impact on student achievement by 
introducing the formal criteria of consistency, cohesion, constancy and control. 
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The consistency principle can be described at the classroom, school and context 
levels. According to this principle, the effectiveness of classrooms, schools and contexts is 
enhanced when the factors at these levels are in line with each other and support each 
other. For example, a math curriculum which offers tests and corrective instruction might 
promote evaluative behaviour of the teacher much stronger than a math curriculum not 
offering these facilities. Although all selected factors at the classroom level are supposed 
to be necessary for effective education, an emphasis on one factor might reduce the impact 
of another factor (Gamoran, 1986). For example, the implementation of a very refined and 
detailed grouping procedure might be so demanding for teachers that the actual time for 
learning in their classrooms is reduced. 

At the school level consistency between the components is an important condition 
for instruction. All members of the school team should take care of that, thereby creating 
cohesion. Creemers (1991) points at the importance of continuity, meaning that schools 
should not change rules and policies every other year. This implies the constancy principle, 
which can be demonstrated in longitudinal research by comparing school level factors from 
year to year. The control principle not only refers to the fact that student achievement 
should be evaluated, but also to the quiet atmosphere in the school. Control also refers to 
teachers holding themselves and others responsible for effectiveness. At the context level 
consistency, constancy and control are again important formal characteristics emphasizing 
the importance of factors over time and of mechanisms to ensure effectiveness. 

Especially the formal criteria were considered a major improvement of the theory 
compared with other models, because these criteria hold together the other factors in the 
model and explain their joint impact. 

3.2 The effects of configurations of school level and classroom level factors on 
student achievement 

Effectiveness research has led to a large set of effective classroom and school factorsall 
supposedly affecting student achievement. Lists of these factors suggest that they are all 
equally important. They do not distinguish between classroom and school factors and they 
do not pay attention to the mutual influence of factors at both levels. The comprehensive 
educational effectiveness model of Creemers (1994) tries to overcome these problems by 
relating the isolated factors to a specific level of effectiveness (i.e. the classroom level, the 
school level, the context level) and to the overarching effectiveness key constructs of 
quality, time and opportunity. Moreover, isolated factors can boost each other’s effects 
when applied together as meaningful configurations. In such a configuration, factors do not 
oppose each other or hinder each other’s effects on student achievement but are in line 
with each other (the consistency principle). 

To test the empirical validity of the effectiveness model, two research questions were 
answered (Reezigt, Guldemond & Creemers, 1997): 

Do the classroom and school factors defined in the model have the expected effects 
on student achievement? 

Is it possible to find configurations of factors at the classroom and the school levels 
which can be considered as examples of the consistency principle, and if this is so, 
what are their effects? 

The factors in the model are expected to have positive effects on achievement. In fact, 
most of the factors were included because they had empirically shown to have positive 
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' effects. However, most effectiveness research so far has not focused on a lot of factors at 
the same time. Most research studies some factors and leaves out most of the others that 
might be important as well. Therefore it is possible that some factors take away each other 
explanatory power when they are analyzed together. As a consequence the absence of 
effects is also considered in line with the expectations of the model. Only negative effects 
of factors are directly opposed to the hypotheses put forward by the model. According to 
the consistency principle, configurations with a higher level of consistency might be 
expected to have more positive effects on student achievement. A consistent configuration 
of factors at the classroom level means that the curriculum, the grouping procedures, and 
the behaviour of the teacher are in line with each other, thereby promoting time for 
learning and opportunity to learn and, in the end, student achievement. A consistent 
configuration of factors at the school level means that the variables measuring quality, 
time and opportunity all go in the same direction and do not reduce each other’s 
effectiveness. 

To answer the first research question, multilevel analysis (VARCL, Longford, 
1988) was applied using three levels (students nested within teachers nested within 
schools). The study included two dependent variables (language achievement and math 
achievement) and three cohorts of elementary school students. To answer the second 
; research question, cluster analysis was performed as a first step to define configurations of 
factors as more or less consistent. Additionally, three-level analysis was performed to test 
the effects of the configurations on student achievement. 

Data initially collected for the evaluation of the Dutch Educational Priority Policy 
were reanalyzed. The data refer to a random sample of elementary schools which is 
representative for the Dutch elementary school population. Repeated measurements took 
place in 1988, 1990 and 1992. A total of 129 schools and 516 teachers (258 from 1990, 
258 from 1992; two teachers per school) were used in the reanalyses. The number of 
students varies from 1531 (one cohort) to 3762 (two cohorts). Students from grades 4, 6 
and 8 (age groups 8, 10 and 12) took several tests in the beginning of the school year. The 
teachers filled in written questionnaires and so did the principals of the schools. Because 
of the repeated measurements, several subsequent cohorts of children in the same schools 
can be studied. One cohort of students has been tested three times during their school 
career: they were in grade 4 in 1988, in grade 6 in 1990 and in grade 8 (the final grade of 
Dutch elementary education) in 1992. All other student cohorts have been tested once or 
twice. 

The dependent variables were math and language achievement. The achievement 
measurement corrected for 2 years earlier refer to two cohorts of students (the students that 
were in grade 4 in 1988 or in 1990 and in grade 6 two years later, and the students that 
were in grade 6 in 1988 or in 1990 and in grade 8 two years later). The achievement 
measurement corrected for 4 years earlier concerns one cohort of students, who were in 
grade 4 in 1988, in grade 6 in 1990 and in grade 8 in 1992. The teacher level was not 
appropriate for these analyses because of nesting problems. 

The independent variables were 11 classroom factors: quality of the curriculum, 

implementation of the curriculum, use of curriculum tests, grouping procedures, home- 
work, clear goal setting, evaluation, feedback, corrective instruction, time for learning and 
opportunity to learn. There were 6 school factors: school rules about classroom instruction, 
evaluation policy, intervision policy (professionalisation of the school team), rules about 
the use of time, orderly atmosphere, rules about the implementation of the school 
curriculum. All factors were directly derived from the Creemers model. 
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Effects of factors 



After controlling for the student level factors small proportions of variance remain at the 
classroom and the school level that may be accounted for by the classroom and school 
factors derived from the model. Table 1 gives an overview of the statistically significant 
regression coefficient estimates of these factors. 

insert Table 1 

Most student factor effects are fairly stable over subjects as well as over the dependent 
variables, except for the gender effects. Prior achievement and intelligence show 
consistently positive effects on achievement, while a lower social class background or an 
ethnic minority background show consistently negative effects. The effects are much 
stronger for achievement over 4 school years than they are for achievement over 2 years. 

Concerning the classroom factors, Table 1 shows negative or mixed (sometimes 
positive, sometimes negative) effects on student achievement of 3 factors: feedback, time 
for learning and opportunity to learn. There are 4 factors that do not have any affect at all: 
use of curriculum tests, grouping procedures, evaluation and corrective instruction. Four 
classroom factors have positive effects: quality of the curriculum, implementation of the 
curriculum, homework and clear goal setting. These positive effects are not very stable 
over effectiveness criteria and school subjects, however. In fact, only homework and clear 
goal setting show stability in the way the effectiveness model assumes. 

Concerning the school factors, the picture is not essentially different: 2 school 
factors (rules about classroom instruction and evaluation policy) have mixed effects, 2 
more school factors have negative effects (policy on intervision and rules about time use) 
and 1 school factor (rules about the implementation of the school curriculum) has no effect 
at all. Only 1 out of 6 school factors shows a positive impact on achievement (orderly 
atmosphere) but this effect is not stable over effectiveness criteria and school subjects. 

To summarize, some factors show expected effects, i.e. either positive effects or no 
effects. The other factors show negative effects, which is contradictory to the expectations 
of the effectiveness model. Even when factors have positive effects, these are not stable 
across analyses, i.e. across cohorts of students and across school subjects. 

Effects of configurations 

The cluster analyses at the classroom level yield five interpretable clusters for language 
and three for mathematics. At the school level three clusters were found. Both at the 
classroom level and at the school level there is always one cluster standing out in a 
negative sense. Teachers and schools in these clusters show consistently low scores on all 
factors. Therefore, these clusters can be considered overall ineffective. 

At the classroom level, 14 % of the teachers in the analysis for language belong to 
the overall ineffective cluster. There are four more language clusters, one of which stands 
out in a clearly positive sense. Teachers in this overall effective cluster (36 %) have high 
scores on more than half of the classroom factors. The teachers in the three remaining 
language clusters have high scores on a limited set of classroom factors only. They focus 
on classroom factors related to grouping (18 %), related to evaluation (4 %) and related to 
homework and clear goal setting (28 % of the teachers). 
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The three math clusters are the overall ineffective cluster (39 % of the teachers), focus on 
instruction (13 %) and focus on opportunity (48 %). Table 2 gives an overview of the 
classroom level clusters. 

insert Table 2 

The school level clusters are the overall ineffective cluster (43 % of the schools), focus on 
rules and order in the school (36 %) and focus on school policy (21 % of the schools). 
Table 3 gives an overview of the school clusters. 

insert Table 3 

According to the consistency principle, factors within levels can be more or less consistent 
and in line with each other. When we apply the consistency principle the overall effective 
cluster (language, classroom level) and the overall ineffective clusters (language and math, 
classroom level and school level) can be seen as consistent in a positive or negative sense. 
The effects of the clusters were contrasted with the effects of the overall ineffective 
classroom and school clusters. Therefore, we expect all effects of the remaining clusters to 
be positive. Furthermore, we expect that the effects of the overall effective clusters are the 
strongest. Table 4 shows the significant regression coefficients. 

insert Table 4 

The findings partly support our expectations. The language clusters show two positive 
effects, caused by the clusters which focus on evaluation and homework/goals. The other 
effects however are not as we expected them to be. The overall effective cluster, the one 
which lives up in a positive sense to the consistency principle, does not differ in its effects 
from the overall ineffective cluster, which can be seen as the dark side of the consistency 
principle. The cluster which focuses on grouping shows negative effects. This implies that 
the effects of this cluster are even more negative than the effects of the overall ineffective 
cluster, the contrast cluster in this analysis. 

The math clusters do not show any effects on student achievement at all. The two 
clusters that should stand out positively when contrasted with the overall ineffective math 
cluster do not differ in their effects on achievement from this contrast cluster. Finally, the 
school level clusters which we expected to show positive effects, instead have negative or 
mixed (sometimes positive, sometimes negative) effects on achievement. The fit of the full 
model, which includes both the classroom and the school clusters, does not improve 
compared to the model which includes the student factors only. Our main conclusion is 
that although we have found clusters that support the consistency principle, the analyses do 
not support our expectation that consistency leads to higher outcomes. 

3.3 Conclusion 

The comprehensive model for educational effectiveness is an example of models that try to 
meet the criteria for theory development within the educational effectiveness area, taking a 
plausible theory about student learning in schools as a point of departure and describing 
factors, formal criteria and relationships in detail. Although the model can be criticised 
(for example, the context level could be elaborated further, the reasons to include or to 



exclude certain factors could be discussed) it is a plausible model based on reviews of 
research. Components of the model are supported by earlier studies. 

The study described above does not support the model very well. There may be 
several reasons. The study merely performed reanalyses on data which were not collected 
specifically to test the model. The data probably are not specific enough to test the theory. 
Also the model itself should be investigated for weaknesses. Other effectiveness models 
are also sometimes supported by research, but not completely. There seems to be a general 
need for further theory development. In the following, our considerations will be related 
and restricted to the comprehensive model of Creemers (1994). We know now that 
educational effectiveness is caused by a multitude of factors, not just by one or a small set. 
The following recommendations can be made for the next future: 

We need more research on the basis of theoretical frameworks that are now 
available. As yet the discrepancy between theoretical insights and the research base 
is too large. We need more sophisticated techniques but most of all we need 
research not restricted to outcomes but incorporating educational processes at the 
classroom and school level. We should not restrict our research programme to 
isolated problems like stability and size of effects or isolated factors. It will be hard 
to test the full model, but at least more levels, more factors and more relationships 
between factors should be pursued in research. 

We need longitudinal studies, experimental studies and case studies. We should not 
only measure the added value of education but also what causes that added value. 
Because we expect that a lot of things go on between pre- and posttest we should 
repeatedly measure what goes on in classrooms and schools. Probably the stability 
of effects is caused by the consistency and constancy over time of the classroom 
and factors. So far we do not have much insight in the role of formal criteria like 
consistency, constancy, cohesion and control of the factors in schools. 

When experimental studies and longitudinal studies are integrated the relationship 
between effectiveness research and educational improvement becomes quite close. 
Preferably school improvement projects should measure what goes on in classrooms 
and schools and they should continue to introduce factors related to effectiveness. 
Probably we have to take our theory somewhat further on by facing the problem 
that teachers, schools and maybe even systems hold opinions and are driven by 
something which makes them more or less willing and capable to pursue 
educational effectiveness. We want to elaborate our theory on the fact that 
educational professionals are more or less willing to pursue effectiveness and to 
maintain factors that have been proven to be effective. 

The next section will discuss the further elaboration of our theory with the introduction of 
visions at the different levels of the educational system in more detail. 

4 Visions that work 

The comprehensive model (Creemers, 1994) contains factors at several levels, it includes 
several components and formal criteria for effectiveness. In this sense the model is really 
what it is called: comprehensive. It summarizes almost everything we can think about 
related to educational effectiveness. However, as yet the model is not very well supported 
by empirical evidence. An explanation could be that the formal criteria of effectiveness 
which deal with questions as to how much in line factors are and how stable they are over 
time are not often found. That could be related to the fact that teachers do not show that 
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kind of behaviour very much, at least not in the time the observer is there or when they 
are answering questions about their behaviour. 

We expect that teachers, school teams and departments who pursue effectiveness 
more consistent, frequent, constant, and permanent achieve higher outcomes. We assume 
that there is a certain drive in teachers, school teams and departments that enforces them to 
pursue effectiveness. Pursuing effectiveness means carrying out effective education by 
providing effective instruction and creating conditions at other levels to enhance effective 
instruction. The model already has some factors related to that drive, like high 
expectations of the teacher and the school mission. The drive that can be found at all 
levels of education is a vision. 

A vision contains presuppositions about life and society in general. More important, 
a vision is an idea about what has to be achieved in a school and how it should be 
achieved based on values and norms. In education, these values and norms are related to 
the aims and functions of the school, and the tasks and positions people have. The concept 
of visions that work relates to the fact that visions, although based on general 
presuppositions about life and values and norms, are not abstract but related to concrete 
behaviour in classrooms and schools. A vision is a value and belief system of teachers 
with respect to their place and function in the educational system combined with the things 
they do based on that vision. A vision is the total of values and norms about education and 
the specific place, tasks, and functions persons have, based on the context they live in and 
related to what they actually do. We expect that almost everybody in the educational 
profession has ideas, opinions, conceptions, and beliefs about education and the objectives 
of education that are part of their belief system and that makes them pursue effectiveness 
more or less strongly. 

There is certainly a lot of research evidence showing that having an opinion does 
not necessarily mean that that opinion leads to a concrete action. There is a difference 
between beliefs and opinions on the one hand and acting on the other hand. The arguments 
and evidence for that discrepancy can be found in schools. Dutch findings of research into 
the development of school working plans where schools were supposed to develop their 
own ideas and objectives about education can illustrate this. There is hardly any 
relationship between school working plans and what actually happens in schools (Van der 
Werf, 1988). There is hardly any relationship between statements about aims and 
objectives and the effectiveness of education. Contra evidence can also be found in the 
progressive schools movement, such as schools based on the ideas of Maria Montessori. 
Sometimes education in these schools is quite effective because the teachers are strong 
believers and pursue their ideas and ideals with great persistence and passion in educa- 
tional practice. 

Aspects of the concept of vision are already included in models of school 
effectiveness. We expect that teacher expectations and beliefs contribute to educational 
outcomes. In the comprehensive model ideas and actual behaviour are intertwined. We can 
improve the model by distinguishing between visions as the total of opinions and beliefs 
about education and the function persons have with respect to that, specific places that can 
motivate them to pursue and to show effective behaviour. Now visions can be subject for 
educational research and the relationships between visions at different levels and what 
actually happens can be investigated. Because past research was directed to actual 
behaviour, factors related to visions could not be found but now, by making this dis- 
tinction, appropriate instruments for the investigation of visions can be developed and the 
concept can be studied more adequately. 
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When we assume that visions can actually contribute to educational effectiveness 
we suppose that there is a relationship between visions, factors and formal criteria for 
effectiveness. Because of this, we have named this section ’Visions that work’. Visions 
that work are the total of ideas, beliefs, and expectations with respect to education of a 
person or a group of persons (departments, schools, districts, nations), the objectives of 
education and the way these should be achieved. Because effectiveness is at stake the 
means of pursuing the objectives and the means to put the beliefs about education into 
practice are the factors and formal criteria related to educational effectiveness. We expect 
that educational effectiveness is more consistent, cohesive, constant and controlled when an 
apprpriate vision provides a basis than when there is no vision at all, or there is a vision 
but it is not or not clearly related to educational effectiveness factors. 

So far there is no direct research evidence for the importance of visions that work at 
various levels, but some circumstantial evidence does exist. 

The classroom level 

We think that teachers who have a vision related to educational effectiveness expect their 
students to learn through education and expect that their instructional procedures contribute 
to student outcomes. They have high expectations about their students, which means that 
they expect them to learn and to achieve. Because they have objectives and aims for their 
own behaviour and for their students they pursue effectiveness. They are constantly 
involved in what contributes in their teaching behaviour and in the instructional processes 
to effectiveness. Teachers are not only aware of the importance of factors related to time 
for learning, opportunity to learn and quality of education, but they are also aware of the 
importance of the formal characteristics like consistency. They expect support from the 
departmental and school level, not only with respect to time, opportunity and quality but 
also in the formal characteristics like consistency, cohesion, constancy and control. They 
expect support from the curriculum, from coaching, in-service training and further pro- 
fessionalisation. They expect educational leadership from department heads and school 
principals. Moreover they expect that they as teachers can contribute to a vision at other 
levels too. This is important because a shared vision is expected to contribute to cohesion 
and constancy. Circumstantial evidence for the importance of visions that work at the 
classroom level can be found in research on high expectations of teachers and their effects 
on teaching and learning processes and outcomes. 

Departmental and school level 

Visions that work at these levels are the shared visions of members of departments or 
school about the objectives of education, the way these objectives should be pursued and 
the means they have to achieve the objectives. We expect teachers as professionals to 
contribute to this shared vision but educational leaders (heads of departments, principals) 
may also take the lead in formulating a shared vision. Visions that work are strongly 
related to what happens in schools and departments and especially what should happen at 
the classroom level. Visions deal with the objectives of education, with opinions of how 
much education and schools can contribute to achieve the objectives. Visions that work 
include clear ideas and opinions about how to pursue objectives within the classroom, 
department and school. 



Negative circumstantial evidence can be found in Dutch research on school 
working plans which were not related to educational practice (Van der Werf, 1988). 
Positive evidence can be found in school work plans in the United Kingdom where ideas 
turned out to be related to educational practice (Hargreaves & Fullan, 1992). Improvement 
projects like Sedherot (Bashi et al., 1990) also prove the importance of shared visions on 
education in relationship to effectiveness factors. These projects have a strong drive to 
pursue educational effectiveness in practice. The ideas developed by Stringfield (1995) 
about high reliability organizations can also be seen as a way to create visions within 
schools to maintain high levels of effectiveness. Unexpected effects of innovations in 
progressive schools or Montessori schools (unexpected because the actual instructional 
behaviour is not always effective) can also be reinterpreted as circumstantial evidence for 
the importance of visions that work. Even when visions are not so strongly related to 
effectiveness factors and sometimes even contrary to it, visions themselves were 
impressive enough to maintain an educational practice with results in children’s behaviour 
or outcomes in schools for a long period of time . 

The levels above the school 

The levels above the school also contain visions, strongly related to ideas about education 
in the society or in a part of the society. These visions concern the importance of educa- 
tion, what education should be like, which objectives should be achieved and how they 
should be achieved. For some empirical evidence we refer to international comparative 
research which shows the importance of visions (Reynolds & Farrell, 1996; Reynolds et 
al., 1997). Because the above school level is more distant from the classroom processes 
visions at thes level deal with topics such as the structure of education, the role of the 
community, the contribution of parents, the relationship between parents and schools, 
national evaluation and monitoring systems and the national curriculum. Countries with a 
strong emphasis on the importance of education, with high expectations about the 
contribution of schools to the achievement of educational goals and the provision of condi- 
tions, the structure and resources for schools to achieve objectives achieve higher 
outcomes. Visions in a specific society are not just ideals about education, but visions that 
work because they are related to measures to be taken by the above school levels and in 
that way related to educational effectiveness at lower levels. 

We have seen that visions have to be related to the factors of educational effectiveness and 
to the formal criteria of effectiveness. Visions emphasize the objectives of education and 
the importance of conditions and processes to achieve those objectives. Visions also 
emphasize the fact that there should be consistency, cohesion, constancy and control. 

Visions that work are not an alternative for the comprehensive model of edu- 
cational effectiveness. They are a further elaboration of the model, not by introducing new 
factors and new formal criteria but by holding together within a level and between levels 
the factors that induce consistency for a longer period of time (constancy) for all members 
of the school team (cohesion) including a control mechanism in order to assure at all 
levels effectiveness over a long period. Vision as a concept is not included in the compre- 
hensive model so far, It is an addition but it does not provide new factors nor new formal 
criteria. We- expect that the concept of vision holds together the factors and criteria that are 
already available in the model. We therefore add vision to the model at different levels 
and connect it with factors such as expectations and mission. 
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Conclusion 



Based on research, theory development, and rethinking of the comprehensive model we 
conclude that a further elaboration of the model is desirable. The concept of vision is 
introduced at the various levels of the model as an overarching factor like time, 
opportunity and quality. The concept of vision combines ideas, opinions, beliefs about 
instruction, schools and education in general. The addition ’that work’ emphasizes that 
visions in educational effectiveness are strongly related to concrete behaviour in the 
classrooms and schools. Vision, in a comprehensive model for educational effectiveness, is 
related to factors for educational effectiveness and the formal criteria. Visions that work 
imply that teachers and others accomplish these factors and criteria. 

This addition has implications for the research programme into educational 
effectiveness. Visions can be included at all levels and induce research into the ideas that 
professionals have about educational effectiveness and how these ideas are related to 
factors and criteria for effectiveness. In this way we can acquire knowledge about why 
educational effectiveness is achieved sometimes but not always. We do not only know 
what happens in education but we also know why it happens. We do not only know what 
schools and classrooms achieve different outcomes but we can also explain what causes 
the differences. As such the concept of vision can contribute to further theory development 
about cause-effects relationships in educational effectiveness and the explanation of these 
relationships. 
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Table 1 



Effects of student, classroom and school level factors on language (L) 
and math (M) achievement; L2 means achievement corrected for prior 
achievement two years earlier, L4 corrected for fours years earlier 
(between brackets for L4 and M4 is explained whether the first or the 
second teacher in the analysis is causing the effect) 



dependent variable 


L2 


L2 


L4 


M2 


M2 


M4 


grand mean 


49.4 


42.3 


57.2 


20.8 


35.8 


34.2 


student level factors 














- prior achievement 


.4 


.4 


.3 


.5 


.5 


.4 


- intelligence 


.2 


.2 


.2 


.2 


.2 


.3 


- gender (girls) 




1.4 




- 2.8 


- 1.1 


- 3.2 


- Dutch lower social class 


- 2.9 




-4.3 


- 2.4 




- 3.4 


ethnic minority 


- 7.7 


-7.3 


-12.3 


- 4.5 


- 2.5 


- 6.1 


classroom level factors 














- quality curriculum 

- implementation curriculum 

- use of curriculum tests 






1.2(2) 






2.1 (2) 


- grouping procedures 

- homework 






1.2 (2) 


.8 






- clear goal setting 

- evaluation 




.7 


1.7 (1) 


.8 




2.5 (1) 


- feedback 

- corrective instruction 






- 2.2 (2) 




1.2 


- 1-8 (2) 


- time for learning 


- .1 




- .1 (1) 








- opportunity to learn 




- .1 


- -1 (2) 


.1 






school level factors 














- rules classroom instruction 


.2 


.4 








- .5 


- evaluation policy 

- intervision policy 


.3 




.4 




- .2 


- .3 


- rules time use 

- orderly atmosphere 

- rules implementation curriculum 


- .5 










.7 


% of variance not accounted for 














- student level 


62 


56 


70 


51 


49 


54 


- classroom level 


3 


5 


n.a. 


4 


4 


n.a. 


- school level 


2 


1 


2 


3 


3 


6 


- total 


67 


62 


72 


58 


56 


60 


improvement of the fit of the model, 
compared to the model including 
only the 4 student factors 


no 


no 


yes 


no 


no 


yes 



0 




21 



23 



Table 2 



Clusters classroom level, language and mathematics 
(based on 11 classroom factors, n=540 teachers) 



clusters language 


% 


number of factors on which teachers in the 




teach. 


clusters achieved a high score 


1. overall effective 


36 


1 (quality curriculum, implementation curriculum, eva- 
luation, feedback, corrective instruction, time for lear- 
ning, opportunity to learn) 


2. focus on grouping 


18 


5 (use of curriculum tests, grouping, feedback, correcti- 
ve instruction, time for learning) 


3. focus on evaluation 


4 


3 (quality curriculum, use of curriculum tests, evaluati- 
on) 


4. focus on homework/goals 


28 


2 (homework, clear goal setting) 


5. overall ineffective 


14 


none 


total 


100 





clusters math 

1. focus on instruction 13 7 (quality curriculum, grouping, homework, evaluation, 

feedback, corrective instruction, time for learning) 

2. focus on opportunity 48 6 (implementation curriculum, use of curricular tests, 

clear goal setting, feedback, corrective instruction, 
opportunity to learn) 



3. overall ineffective 



39 



none 



Table 3 



Clusters school level (based on 6 school factors, n=135 schools) 





% 


number of factors on which the schools 




sch. 


achieved a high score 


1. focus on rules/order 


36 


3 (rules instruction, rules time, orderly atmosphere) 


2. focus on school policy 


21 


3 (evaluation policy, professionalisation policy s, 

implementation curriculum) 


3. overall ineffective 


43 


none 


total 




100 




23 



Table 4 



Fffects of classroom and school clusters on language and math achievement 
(betH^n brackets for L4 and M4 is explained whether the Ik. or the second teacher 

in the analysis is causing the effect) 



dependent variable 
grand mean 

classroom level: clusters language 



L2 L2 L4 
41.6 38.4 53.2 



M2 M2 M4 
34.7 27.4 43.9 



- 1 overall effective 

- 2 focus on grouping 

- 3 focus on evaluation 

- 4 focus on homework/goals 

classroom level: clusters math 

- 1 focus on instruction 

- 2 focus on opportunity 

school level 

- 1 focus on rules/order 

- 2 focus on school policy 

% of variance not accounted for 

- student level 

- classroom level 

- school level 

- total 

improvement of the fit of the model, 
compared to the model including 
only the 4 student factors 



2.9 



- 4.6 (2) 
2.4(1) 



1.6 



62 56 70 

3 5 n.a. 

2 1 4 

67 62 74 



no no no 



- 3.8 
2.2 -2.9 



51 49 55 

4 4 n.a. 

4 3 10 

59 56 65 



no no no 
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24 



levels 



components/characteristics of quality, time and opportunity 



formal 

criteria 



criteria for 
effectiveness 



school 



classroom 



quality 


* policy focussing on effectiveness 

* indicator system/policy on evaluation/national testing 
system 

* training and support system 

* funding based on outcomes 




consistency 


time * national guidelines for time schedules 

* supervision of time schedules 

opportu- * national guidelines for curriculum 
mtv 




constancy 

control 


> 


r 






quality/ * rules and agreements about classroom instruc- 
educa- tion 

tional * evaluation policy/evaluation system 




consistency 


quality/ * policy on intervision, supervision, professiona- 
orgamza- lization 

tional * school culture inducing effectiveness 




cohesion 


time 


* time schedule 

* rules and agreements about time use 

* orderly and - quiet atmosphere 




constancy 


oppor- 

tunity 


* school curriculum 

* consensus about mission 

* rules and agreements about how to implement the 
school curriculum 




control 


i 


' 







quality of instruction 
curriculum 

* explicitness and ordering of goals and content 

* structure and clarity of content 

* advance organizers 

* evaluation 

* feedback 

* corrective instruction 
grouping procedures 

* mastery learning 

* ability grouping 

* cooperative learning 
highlv dependent on 

* differentiated material 

* evaluation 

* feedback 

* corrective instruction 
teacher behaviour 

* management/orderlv and quiet atmosphere 

* homework 

* high expectations 

* clear goal setting 

- restricted set of goals 

- emphasis on basic skills 

- emphasis on cognitive learning and transfer 

* structuring the content 

- ordering of goals and content 

- advance organizers 

- prior knowledge 

* clarity of presenfation 

* questioning 

* immediate exercise 

* evaluation 

* feedback 

* corrective instruction 




time for learning 
opportunity to learn 



student 



time on task 
opportunities used 



• basic skills 

• higher order 
skills 

• metacogniti- 
ve skills 

* — 



motivation 

-XT 
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aptitudes 

social background 



Figure 1 A comprehensive model of educational effectiveness 

(Creemers, 1994) ^ BE 
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